Skip to content

draft changes to cs-api for DB based provider#133

Merged
grahamegrieve merged 13 commits intomainfrom
2026-02-gg-cs-api-proposal
Feb 23, 2026
Merged

draft changes to cs-api for DB based provider#133
grahamegrieve merged 13 commits intomainfrom
2026-02-gg-cs-api-proposal

Conversation

@grahamegrieve
Copy link
Contributor

No description provided.

@grahamegrieve grahamegrieve marked this pull request as draft February 21, 2026 06:00
jmandel added a commit to jmandel/FHIRsmith that referenced this pull request Feb 21, 2026
…ffset/filterExclude

Add Grahame's proposed decomposed filter API from PR HealthIntersections#133 alongside
existing expandForValueSet. The new API pushes excludes and offset/count
through the filter pipeline:

cs-api.js:
  - handlesExcludes() / handlesOffset() capability queries
  - getPrepContext(iterate, offset, count) with paging params
  - filterExclude(ctx, filters[]) for exclude accumulation

cs-sqlite-runtime-v0.js:
  - Implements all new API methods
  - filterExclude validates filters, throws for unsupported
  - executeFilters builds combined SQL with excludes + offset/count
    when accumulated, falls back to normal filter sets otherwise
  - Reuses #buildV0FilterSql / #buildV0ExcludeNotExists for SQL generation

expand.js (worker):
  - scanValueSet() detects simple single-CS value sets
  - excludeFilterList() converts exclude clauses to {prop, op, value}
  - includeCodes passes excludes via filterExclude when CS handles them
  - Graceful fallback: if filterExclude throws, clears accumulated
    excludes and worker-side excludeCodes handles it

All 22 cross-system tests pass with same results and performance.
52 pre-existing jest test failures unrelated to these changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jmandel added a commit to jmandel/FHIRsmith that referenced this pull request Feb 21, 2026
Based on Grahame Grieve's PR HealthIntersections#133: decomposed CodeSystem provider API into a
filter pipeline (filter → filterExclude → executeFilters → filterMore/filterConcept).

Adds a unified SQLite-based CodeSystem provider (cs-sqlite-runtime-v0) that can
serve any terminology imported into the v0 schema, replacing the need for
per-terminology custom providers. Includes specialized SNOMED CT support for
expressions, ECL, and hierarchy queries.

Key features:
- Single combined SQL query for all include filters, exclude filters, concept
  lists, and concept excludes (via executeFilters intent pipeline)
- Batch designation pre-fetch via new prepareDesignations() API, eliminating
  N+1 per-code DB queries during iteration
- LIMIT/OFFSET passdown to SQL for all providers (not just simple ValueSets)
- REGEXP filter support with effort-based query breaker
- Importers for RxNorm, LOINC, and SNOMED CT into v0 schema

New CS provider API methods (backward-compatible):
- includeConcepts(filterContext, codes) — batch concept registration
- excludeConcepts(filterContext, codes) — clean concept exclusion
- locateBatch(codes, filterSet) — batch context lookup
- prepareDesignations(filterContext, options) — designation pre-fetch hints

Worker changes:
- Unified concept + filter intent block (one executeFilters call)
- Skip redundant excludeCodes() when provider handles excludes in SQL
- LIMIT passdown via cs.handlesOffset() for cross-system ValueSets
- prepareDesignations() call before executeFilters()

Performance (v0 provider, full expansion, no count limit):
- RxNorm SBD+SCD (27K codes): 0.5s
- SNOMED Clinical Findings (124K codes): 2.4s
- SNOMED All Procedures (59K codes): 0.9s
- IPS Procedures with 8 excludes: 53ms (count=100)
- Cross-system RxNorm+LOINC: 21ms (count=10, was 1800ms)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jmandel added a commit to jmandel/FHIRsmith that referenced this pull request Feb 22, 2026
Add SQLite-backed code system providers for RxNorm, LOINC, and SNOMED CT
that use a shared v0 schema with closure tables, FTS5 search indexes,
and a unified SQL filter pipeline for both includes and excludes.

Key features:
- Single #buildV0FilterSql code path handles all filter types (concept
  hierarchy, property filters, code regex, value set membership)
- Excludes reuse the same filter SQL wrapped in NOT EXISTS
- Streaming pagination for large expansions (124K+ SNOMED codes)
- Batch designation fetching for efficient display/property loading
- SNOMED expression constraint language support via adapter
- RxNorm archived concept import from RXNATOMARCHIVE
- STY registered as filterable property for RxNorm
- Opt-in perf counters (no-op when disabled)

Integrates with Grahame's CS provider API (PR HealthIntersections#133):
- getPrepContext, filterExcludeFilters, filterExcludeConcepts
- scanValueSet, handlesExcludes, handlesOffset
- Unified intent path with includeConcepts + filter + exclude

Also fixes method name bugs on legacy filter path (getExtensions ->
extensions, getCodeStatus -> getStatus, getProperties -> properties).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jmandel added a commit to jmandel/FHIRsmith that referenced this pull request Feb 22, 2026
Add SQLite-backed code system providers for RxNorm, LOINC, and SNOMED CT
that use a shared v0 schema with closure tables, FTS5 search indexes,
and a unified SQL filter pipeline for both includes and excludes.

Key features:
- Single #buildV0FilterSql code path handles all filter types (concept
  hierarchy, property filters, code regex, value set membership)
- Excludes reuse the same filter SQL wrapped in NOT EXISTS
- Streaming pagination for large expansions (124K+ SNOMED codes)
- Batch designation fetching for efficient display/property loading
- SNOMED expression constraint language support via adapter
- RxNorm archived concept import from RXNATOMARCHIVE
- STY registered as filterable property for RxNorm
- Opt-in perf counters (no-op when disabled)

Integrates with Grahame's CS provider API (PR HealthIntersections#133):
- getPrepContext, filterExcludeFilters, filterExcludeConcepts
- scanValueSet, handlesExcludes, handlesOffset
- Unified intent path with includeConcepts + filter + exclude

Also fixes method name bugs on legacy filter path (getExtensions ->
extensions, getCodeStatus -> getStatus, getProperties -> properties).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
jmandel added a commit to jmandel/FHIRsmith that referenced this pull request Feb 22, 2026
Add SQLite-backed code system providers for RxNorm, LOINC, and SNOMED CT
that use a shared v0 schema with closure tables, FTS5 search indexes,
and a unified SQL filter pipeline for both includes and excludes.

Key features:
- Single #buildV0FilterSql code path handles all filter types (concept
  hierarchy, property filters, code regex, value set membership)
- Excludes reuse the same filter SQL wrapped in NOT EXISTS
- Streaming pagination for large expansions (124K+ SNOMED codes)
- Batch designation fetching for efficient display/property loading
- SNOMED expression constraint language support via adapter
- RxNorm archived concept import from RXNATOMARCHIVE
- STY registered as filterable property for RxNorm
- Opt-in perf counters (no-op when disabled)

Integrates with Grahame's CS provider API (PR HealthIntersections#133):
- getPrepContext, filterExcludeFilters, filterExcludeConcepts
- scanValueSet, handlesExcludes, handlesOffset
- Unified intent path with includeConcepts + filter + exclude

Also fixes method name bugs on legacy filter path (getExtensions ->
extensions, getCodeStatus -> getStatus, getProperties -> properties).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@grahamegrieve grahamegrieve marked this pull request as ready for review February 23, 2026 10:23
@grahamegrieve grahamegrieve merged commit 45fac19 into main Feb 23, 2026
5 of 7 checks passed
@grahamegrieve grahamegrieve deleted the 2026-02-gg-cs-api-proposal branch February 23, 2026 10:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant